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^ ■ Neural networks are supposed to recognise blurred images (or patterns) of pixels (bits) each. 
^ Application of the network to an initial blurred version of one of P pre-assigned patterns should 
^ . converge to the correct pattern. In the "standard" Hopfield model, the "neurons" are connected to 
O [ each other via A^^ bonds which contain the information on the stored patterns. Thus computer time 
I— I and memory in general grow with A^^. The Hebb rule assigns synaptic coupling strengths proportional 
to the overlap of the stored patterns at the two coupled neurons. Here we simulate the Hopfield model 
J> • on the Barabasi- Albert scale-free network, in which each newly added neuron is connected to only m 
. other neurons, and at the end the number of neurons with q neighbours decays as Although the 

\^ [ quality of retrieval decreases for small m, we find good associative memory for 1 <C m <C A^. Hence, 

I these networks gain a factor N/m ^ 1 in the computer memory and time. 
^ '■ PACS: 05.40-a, 05.50+q, 87.18. Sn 



Traditional neural network models have nodes i ("neurons") coupled to all other nodes 
k with some coupling constant Jik ("synaptic strength"), similar to Sherrington-Kirkpatrick 
infinite-range spin glasses POj. Here we consider one of the simplest neural network models, due 
to Hopfield [2^. This model was mostly applied to infinite range and was only rarely put onto 
Q _ a square lattice with short-range interactions [Sll^. Real neural networks seem to have neither 
O ■ infinite nor only nearest-neighbour connections. The spatial structures of neural networks were 
^ ■ investigated [5] and compared with smaU- world and scale- free networks El IHl El UDj • Now we 
^ present computer simulations of the Hopfield model with Hebb couplings between neighbours 
restricted to a Barabasi- Albert (BA) scale- free network 

In the Hopfield model, each of N neurons or sites can be firing {Si = +1) or not firing 
{Si = —1). Neurons are coupled through Jik, and are sequentially updated according to 

Si ^ sign{J2JikSk) ■ (1) 

k 
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(We mostly ignore the diagonal terms i = k in our sums.) This rule corresponds to a low- 
temperature Monte Carlo simulation of a spin glass. The model has stored P different patterns 
(^f (/X = 1,2 ... ,P), which we take as random strings of ±1. The couplings are given by the 
Hebb rule: 

J^k = T.^^e, . (2) 

The first of these patterns is presented to the network in a corrupted form Si, with ten percent 
of the Si reversed in comparison to the correct The question is whether the iteration through 
Eq. (1) transforms the erroneous Si into the correct ^l. The quality of this pattern recognition 
is given by the overlap 

<iI = Y^S.^l/N , (3) 

i 

which is related to the Hamming distance and equals 1 for complete recognition and ~ ±1/\/N 
for only accidental agreement at random sites; it is ~ 0.8 at the beginning of the pattern 
recognition process, due to the ten percent reversal. 

Now we restrict the synaptic connections Jik to neurons which are neighbours in the BA 
network, but we still use Eqs. (1-3). In these networks, we start from a small core of m sites, 
all connected with each other. Then N ^ m nodes are added, one after the other. Each new 
site i selects exactly m sites from the already existing network sites as its neighbours k, with 
a probability proportional to the number of neighbours which the existing site k has already: 
The rich get richer. When the network has added sites with a total of + m sites, its 
growth is stopped and the neural process of Eqs. (1-3) starts. Synaptic connections Jik exist 
only between sites i and k which are neighbours. 

Since no longer every neuron is connected to all other neurons, the memory-saving trick of 
Penna and Oliveira JT] to avoid storing the Jik no longer applies. 400 Megabyte were needed 
for A^ = 10, 000 nodes and P = 20, 000 patterns. To save computer time, the Jik should be 
determined after and not before the construction of the BA network. 

When only one pattern is stored, it is recognised completely after two iterations. With 
P > 1, however, no complete recognition takes place, the overlap \1/ is usually at the final fixed 
point (reached after about five iterations) lower than at the beginning, as shown in Fig. la . 
However, the model can still recognise the first pattern as the one presented to it, since the 
overlap \1/ ~ 0.19 for P = N = 10^ is still appreciably larger than the overlap l^&l < 500 with 
the other (P — 1) patterns. 

Rather similar results are obtained if we work on a nearest-neighbour hypercubic lattice 
with N = L'^ sites, similar to the studies made in [311] in two dimensions. Fig. lb shows that 
only for small numbers P of patterns an increased d means an increased final overlap. For 
d = 7, 10 and 15 the overlaps with up to 20 patterns did not differ appreciably from d = 5. No 
significant size effects were seen for 4 < L < 20 in five and 4 < L < 13 in seven dimensions. 
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Fig. 1 is based on one sample only for each point; using instead 100 samples at m = 3 and 

= 10,000, we see in Fig. 2 that the overlap varies roughly as ^(P) — 0.19 oc P~°'^, except 
for very small m. A similar power law P~^-^ is also found for hypercubic lattices (not shown.) 
It would be interesting to understand this power law from some analytical analysis. 

A much better recovery of the corrupted pattern is obtained if we take a larger inner core of 
the BA network, that means if m is no longer small. (The first m network sites are all mutually 
connected, as in the traditional Hopfield model.) Using 100 patterns. Fig. 3a shows the overlap 
for N + m = 10^ total sites as a function of m. Already at m = 200, A^ = 9800 the final 
overlap is 88 percent; at m = 2000, A^ = 8000 we have complete recovery. For 1 <^ m -C A^ 
the number of connections (counting each bond twice) is mN in our case and A^^ in the fully 
connected case; thus we saved connections by a factor m/N. If we include the diagonal terms 
J{i, i) = P in Eq. (1), we prevent the overlap from becoming worse than the initial overlap 0.8 
for small m and still get overlap near unity for large m. Fig. 3b. 

Of course, with a large m the network is no longer scale free, as shown in Fig. 4: The simple 
power law oc 1/g'^ for the number of sites with q neighbours each jH] persists for 10^ < q < 10^, 
but a Gaussian peak is added for large q. However, the additional bump concerns only a 
relatively small number of sites, and is probably negligible for any practical purposes. 

For infinite range, m = N, the usual Hopfield model \VI] gives an overlap close to 1 for 
P/N < 0.14 and a relatively small overlap \E' ~ 0.2 for P/N > 0.14, with a sharp jump at 
P/N = 0.14. Our simulations, in contrast, show a gradual deterioration as soon as more than 
one pattern is stored, but the value of \E' is still of order 0.2 and distinctly larger than for the 
other (P — 1) patterns. Using a medium-sized fully connected core, like m ~ 10'^ at P ~ 10^, 
surrounded by a larger BA network with A^ ~ 10^ sites, gives a good compromise between good 
recovery and not too many connections. 

So far it is not clear if the efficient recovery simply results from the relatively large average 
coordination number m, or by some additional ingredients in the problem. It would also be 
interesting if Nature takes advantage of a similar efficiency. If it does, do natural neural networks 
share some geometrical features with the large-m (but finite) scale- free networks 0? 

We thank D. Horn for a discussion and the German-Israeli Foundation for supporting our 
collaboration. 
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Final overlap (recovered bits), for Hopfleld-Barabasi-Albert, N = 10,000, one sample, m=2,3,5 
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Figure 1: Final overlap ^ as a function of the number P of patterns, for N ~ 10^ neurons. 
Each point is based on one sample only. Part a: BA network with m = 2, 3 and 5 from bottom 
to top. Part b: Nearest-neighbour hypercubic lattice in one to five dimensions as shown in 
headhne. 
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Final overlap - 0.1 9, for 1 0,000 neurons, 1 00 samples, m=3; slope -0.6 
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Figure 2: Approximate power law variation of final overlap difference (\E' — 0.f9) with the 
number of patterns P, averaged over 100 samples, with N = 10^ at m = 3. The straight line 
has a slope —0.6. 
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Transition from bad (iow m) to good (high m) behaviour; N = 10000-m; P = 10, 100, 1000 from ieftto right 




Same with diagonal term included In Eq. (1); N = 10000 - m, P = 10, 100, 1000 from ieftto right 




Figure 3: Variation of final overlap (not normalized) with the size m of the fully connected 
core, surrounded hy N — 10^ — m BA sites having m neighbours each, at P = 10, 100 and 
1000 (from left to right). Already for P <^ m <^ N the corrupted pattern is restored well. The 
lowest data points refer to P = 100, N = 3000 — m. Part a ignores the diagonal term in the 
sum (1), while part b includes it. 
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Number of sites having tlie given number of neiglnbours, m = 2 (left) and 100 (right) 
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Figure 4: Number of sites having q neighbours for = 10^, summed over 100,000 simulations, 
at m = 100 (right data). We no longer get the simple power law const/g'^, shown here for 
comparison at m = 2 (left data). 
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